Automated food selection using hyperspectral sensing

ABSTRACT

A solution for automated food selection includes: an order processing component operable to receive an order identifying an item; a selection component comprising: a first hyperspectral sensor operable to sense a reflection from the item and produce a sensor output based at least on the reflection; and a picking mechanism; a control component operable to: based at least on identification of the item, select a hyperspectral profile from a set of hyperspectral profiles; compare the sensor output with the selected hyperspectral profile; and based at least on the comparison, determine whether to select the item for fulfillment of the order, wherein the picking mechanism is operable to divert the item to a selection output based at least on a determination to select the item for fulfillment of the order; and a transport component operable to transport the item to an order storage zone.

BACKGROUND

Selecting perishable food items, such as produce, for customer orderfulfillment typically requires human labor. This is because, even if theperishable food items had passed inspection upon arrival at theorder-fulfilling facility (such as a retail facility that delivers orprovides curbside pick-up), the condition of the perishable food itemsmay change between arrival and ultimate delivery to a customer. Althoughthe perishable food items may have been fresh or even in a pre-ripestate, multiple factors, including time and environmental conditions(temperature and humidity), may impact the rate of spoilage while storedin inventory locations or a staging location for customer pick-up.Humans that are trained to properly ascertain the state of perishablefood items are thus used to pick specific ones of the perishable fooditems for order fulfillment. Any limitations on the human workforce,such as training, illness, and other scheduling issues may thennegatively impact order fulfillment capacity.

SUMMARY

The disclosed examples are described in detail below with reference tothe accompanying drawing figures listed below. The following summary isprovided to illustrate some examples disclosed herein. It is not meant,however, to limit all examples to any particular configuration orsequence of operations.

A solution for automated food selection includes: an order processingcomponent operable to receive an order identifying an item; a selectioncomponent comprising: a first hyperspectral sensor operable to sense areflection from the item and produce a sensor output based at least onthe reflection; and a picking mechanism; a control component operableto: receive the order from the order processing component; receive thesensor output from the selection component; based at least onidentification of the item, select a hyperspectral profile from a set ofhyperspectral profiles; compare the sensor output with the selectedhyperspectral profile; and based at least on the comparison, determinewhether to select the item for fulfillment of the order, wherein thepicking mechanism is operable to divert the item to a selection outputbased at least on a determination to select the item for fulfillment ofthe order; and a transport component operable to transport the item toan order storage zone.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed examples are described in detail below with reference tothe accompanying drawing figures listed below:

FIG. 1 illustrates an exemplary arrangement that advantageously useshyperspectral sensing in automated food selection (e.g., selection ofperishable food items for automated order fulfillment);

FIGS. 2A, 2B, and 2C show further detail for components of thearrangement of FIG. 1;

FIG. 3 illustrates notional hyperspectral selection criteria for use inthe arrangement of FIG. 1;

FIGS. 4A and 4B illustrate exemplary options for implementing aspects ofthe arrangement of FIG. 1;

FIG. 5 illustrates an exemplary automated storage and retrieval system(ASRS) that may be used in the arrangement of FIG. 1;

FIG. 6 shows a flow chart of exemplary operations associated with thearrangement of FIG. 1;

FIG. 7 shows another flow chart of exemplary operations associated withthe arrangement of FIG. 1; and

FIG. 8 is a block diagram of an example computing node for implementingaspects disclosed herein.

Corresponding reference characters indicate corresponding partsthroughout the drawings. Elements in the figures are illustrated forsimplicity and clarity and have not necessarily been drawn to scale. Forexample, the dimensions and/or relative positioning of some of theelements in the figures may be exaggerated relative to other elements tohelp to improve understanding. Also, common but well-understood elementsthat are useful or necessary in a commercially feasible embodiment maynot be depicted, in order to facilitate a less obstructed view.

DETAILED DESCRIPTION

A more detailed understanding may be obtained from the followingdescription, presented by way of example, in conjunction with theaccompanying drawings. The entities, connections, arrangements, and thelike that are depicted in, and in connection with the various figures,are presented by way of example and not by way of limitation. As such,any and all statements or other indications as to what a particularfigure depicts, what a particular element or entity in a particularfigure is or has, and any and all similar statements, that may inisolation and out of context be read as absolute and therefore limiting,may only properly be read as being constructively preceded by a clausesuch as “In at least some embodiments, . . . ” For brevity and clarity,this implied leading clause is not repeated ad nauseum.

Aspects of the disclosure operate in an unconventional way to sense ahyperspectral a reflection from an item (e.g., a perishable food item),compare sensor output with a selected hyperspectral profile, anddetermining whether to select the item for fulfillment of an order. Forexample, a solution for automated food selection includes: an orderprocessing component operable to receive an order identifying an item; aselection component comprising: a first hyperspectral sensor operable tosense a reflection from the item and produce a sensor output based atleast on the reflection; and a picking mechanism; a control componentoperable to: receive the order from the order processing component;receive the sensor output from the selection component; based at leaston identification of the item, select a hyperspectral profile from a setof hyperspectral profiles; compare the sensor output with the selectedhyperspectral profile; and based at least on the comparison, determinewhether to select the item for fulfillment of the order, wherein thepicking mechanism is operable to divert the item to a selection outputbased at least on a determination to select the item for fulfillment ofthe order; and a transport component operable to transport the item toan order storage zone.

Aspects of the disclosure provide a technical solution that improvesoperational efficiency by permitting automated picking of perishablefood items (e.g., produce items such as fruits and vegetables). Thispermits automating (at least partially) order fulfillment centers forhigher operational volumes.

FIG. 1 illustrates an exemplary arrangement 100 that advantageously useshyperspectral sensing in automated food selection (e.g., selection ofperishable food items for automated order fulfillment). For example,arrangement 100 permits a customer 102 to place an order 104 for an item120 (e.g., a perishable food item) over a network 830, and then pick up(retrieve) a suitably selected item 120 s (selected version of item 120)from an order storage zone 500, which may be an automated storage andretrieval system (ASRS) 500 a (described in relation to FIG. 5).

Components of arrangement 100 are described individually in relation toeach of FIGS. 1-5, and the interactions among the individual componentsare described in further detail following the description of FIG. 5.Arrangement 100 includes an order processing component 110, whichreceives an order 104 from a customer 102 over a network 830. In someexamples, order processing component 110 is implemented on a computingnode 800 or a cloud resource 828. Network 830, computing node 800, andcloud resource 828 are described in further detail in relation to FIG.8.

Order 104 may include an item identification 106 that specifies item 120(e.g., banana, apple, strawberry, etc.) and a quantity value 108 thatspecifies the count of items 120 desired. Order 104 may also includepreferences such as a ripeness specification (e.g., rime now or ripe intwo days), and a time estimate 114 of when customer 102 expects toretrieve item 120. In some examples, if hyperspectral sensing permits anestimate of how long item 120 has remaining prior to perishing, timeestimate 114 may be used to select or reject a candidate item 120. At alater time, after retrieving item 120 s from order storage zone 500,customer 102 may provide customer feedback 116, identifying whether item120 met expectations or had already perished. In some examples, customerfeedback 116 may be used to update selection criteria for arrangement100.

Order processing component 110 forwards order 104 to a control component250, which may also be implemented as a computing node 800 or a cloudresource 828. Control component 250 also receives sensor output 206,which includes hyperspectral sensing of item 120, and uses a selectionlogic 260 to determine whether to select item 120 to fulfil order 104,or to reject item 120. Control component 250 leverages a data store 270to support selection logic 260. Control component 250 is described infurther detail in relation to FIG. 2B, and data store 270 is describedin further detail in relation to FIG. 2C.

The action of selecting item 120 for order 104 or rejecting (notselecting) item 120 is performed by a selection component 200. Item 120is transported among selection component 200, an inventory zone 118,which holds inventory of items 120 and other items that may be processedfor selection or rejection by selection component 200, order storagezone 500, and a disposal zone 130. Transportation is provided by atransport component 400, which is illustrated in FIG. 4, and includes atransport vehicle 122 a, a transport vehicle 122 b, and a transportvehicle 122 c. In some examples, inventory zone 118, selection component200, order storage zone 500, and disposal zone 130 are in fixedlocations, and transport vehicles 122 a-122 c move item 120. In someexamples, one or more of inventory zone 118, selection component 200,order storage zone 500, and disposal zone 130 are immediately adjacent,so item 120 is moved via robot arms or conveyor belts rather thanvehicles. In some examples, one or more of inventory zone 118, selectioncomponent 200, order storage zone 500, and disposal zone 130 is mobileand when item 120 is moved, it is moved within a mobile one of inventoryzone 118, selection component 200, order storage zone 500, or disposalzone 130.

In some examples, item 120 is moved in container 124 a, container 124 b,or container 124 c. For example, container 124 a may be an inventorytote, used for ferrying inventory to and from shelves in inventory zone118. Container 124 b may be a customer order tote, used for ferryingitem 120 to order storage zone 500. Container 124 c may be a disposaltote, used for ferrying potentially spoiled produce to disposal zone130.

In some examples, selection component 200 includes a plurality ofselection stations, for example, selection station 202, selectionstation 202 a, selection station 202 b, and selection station 202 c. Asillustrated, selection station 202 includes a hyperspectral sensor 204(which provides at least a portion of sensor output 206), a pickingmechanism 220. Picking mechanism 220 moves (diverts) item 120 to one ofa selection output 230, a disposal output 232, and an inventory returnoutput 234. If item 120 is diverted to selection output 230, item 120will be transported to order storage zone 500. If item 120 is divertedto disposal output 232, item 120 will be transported to disposal zone130. If item 120 is diverted to inventory return output 234, item 120will be transported back to inventory zone 118. In some examples,selection stations 202 a are similarly equipped. Selection component 200is described in further detail in relation to FIG. 2A.

A human inspector 132, or a more finely-trained automatic inspectionsolution, performs a detailed inspection of item 120 d, which is aversion of item 120 that has been sent to disposal zone 130. Humaninspector 132 makes a disposition determination for item 120 d, such asdonate, offer for reduced price, or throw into the trash. Humaninspector 132 also provides feedback using terminal 134, for examplewhether the determination made by control component 250 was correct.Control component 250 is then able to use this feedback to improve thequality of future determinations regarding ripeness and suitability forsale of perishable food items.

FIG. 2A shows further detail for selection component 200. Selectionstation 202, of selection component 200. As illustrated, selectionstation 202 includes two hyperspectral sensors 204 and 204 a, although,in some examples, a different number are used. Hyperspectral sensors 204and 204 a provide at least a portion of sensor output 206. An opticalcamera 208 also provides a portion of sensor output 206. Some examplesuse a light source 210 that provides a calibrated spectrum so that areflection 212 from item 120 has the proper spectral properties in orderto make a proper determination of whether to select item 120. That is,in some examples, light source 210 and hyperspectral sensor 204 arecalibrated (one or the other, or both) to produce a predetermined sensoroutput for a calibration standard. In this way, sensor output 206 willhave the proper spectral properties when compared with known ripenessconditions. Hyperspectral sensor 204 a and optical camera 208 may besimilarly calibrated with light source 210 or another light source.

In some examples, a barcode 214 accompanies item 120, either affixed toitem 120 or a container holding item 120. A barcode reader 216 readsbarcode 214 to identify item 120. The data read from barcode 214 may beadded to sensor output 206 for forwarding to control component 250.Picking mechanism 220 moves item 120 to one of selection output 230,disposal output 232, and inventory return output 234, and may place item120 into one of containers 124 a, 124 b, or 124 c. In some examples, ahuman operator 222 actually performs the picking, receiving instructionsfrom control component 250 over a terminal 224.

FIG. 2B shows further detail for control component 250. Controlcomponent 250 has a profile selection component 252 that selects ahyperspectral profile to be selected hyperspectral profile 254, which isused for making the determination of whether to select item 120 tofulfil order 104. Further detail on hyperspectral profiles is providedin relation to FIG. 3. Selected hyperspectral profile 254 is chosen(selected) based on information within order 104, for example item type(apple, banana, strawberry, or other) and ripeness preference which isindicated in customer preferences 256. Selection logic 260 uses selectedhyperspectral profile 254 along with sensor output 206 to make thedetermination. Control component 250 may also have a profile updatecomponent 262 that uses feedback 264 to improve predeterminedhyperspectral profiles stored in a set of hyperspectral profiles 274(shown in FIG. 2C). Any of profile selection component 252, selectionlogic 260, and profile update component 262 may use or include at leasta portion of a machine learning (ML) component 266. As used herein MLincludes both ML and artificial intelligence (AI).

FIG. 2C shows further detail for data store 270. Data store 270 includesset of hyperspectral profiles 274 that may be indexed based on itemtype, supplier, sensed conditions (see FIG. 3) and ripeness estimates.In some examples, ripeness estimates include not yet ripe, ripe, andexpired (perished). In some examples, ripeness estimates include thenumber of days until expiration. As illustrated, data store 270 alsoincludes customer data 276, which includes order 104, a customer profile278, and customer feedback 116. In some examples, customer profile 278includes order histories and customer preferences that indicate whetherthe customer has a preference for items from a specific supplier or theaverage delay between placing an order an retrieving it (which may beused to set time estimate 114 of FIG. 1). In some examples, data store270 also includes historical data 280 of prior item selections andfeedback, industry data 282 including supplier information and foodquality standards, and localization data 284 that may include the speedwith which produce ripens and expires in the locality of arrangement100. For example, certain perishable foods may expire in warmer climatesduring the summer than in colder climates during the winter.

FIG. 3 illustrates a notional hyperspectral plot 300 that forms thebasis for hyperspectral selection criteria, in some examples ofarrangement 100. Plot 300 shows intensity of reflection 212, against avertical value axis 302, in various spectral bands, spread across ahorizontal frequency axis 304. An optical range 306 is annotated on plot300 to illustrate that sensed reflection 212 has hyperspectral frequencycomponents outside of optical light frequencies. Five spectralcomponents are illustrated, spectral component 310, spectral component312, spectral component 314, spectral component 316, and spectralcomponent 318, although it should be understood that a different numberof spectral components (possibly hundreds or more) may be used. A lowerthreshold 320 and an upper threshold 322, which each varies by spectralband, are illustrated. In some examples, if each of spectral components310-318 falls within lower threshold 320 and upper threshold 322, item120 may be selected to fulfil order 104. In some examples, selectionlogic 260 uses more complex criteria, such as the value of one ofspectral components 310-318 affects the range of acceptability (minimumand maximum value) for another one of spectral components 310-318.

FIG. 4A illustrates an option for implementing arrangement 100 using astationary selection component 200. As illustrated in FIG. 4a , each ofinventory zone 118, selection component 200, order storage zone 500, anddisposal zone 130 are in fixed locations. Transport component 400includes transport vehicle 122 and transport vehicles 122 a-122 c.Transport vehicle 122 is shown as holding item 120 in a container 124,which may be one of containers 124 a-124 c of FIG. 1, or a similarcontainer. FIG. 4B illustrates another option for implementingarrangement 100 using a mobile selection component 200. In thisillustrated option, selection station 202 moves around among inventoryzone 118, order storage zone 500, and disposal zone 130, carrying item120.

FIG. 5 illustrates ASRS 500 a, which may be an implementation of orderstorage zone 500. ASRS 500 a has a dispensing aperture 502, a userdisplay screen 504, and a sensor 506, which may be a two-dimensional(2D) barcode reader. Item 120 s (e.g., item 120 after being selected fororder 104) is contained within container 124 b in a storage location 510(e.g., a shelf compartment) within ASRS 500 a.

To retrieve item 120, customer 102 brings customer's mobile device 520,showing a 2D barcode 522 to be read by sensor 506. Optionally, userdisplay screen 504 is a touchscreen and customer 102 requests item 120 avia interaction with user display screen 504. ASRS 500 a repositionscontainer 124 b to dispensing aperture 502, where customer 102 is ableto pick it up. Note that FIG. 5 does not show ASRS 500 a and mobiledevice 520 at proper relative scale. ASRS 500 a is larger than mobiledevice 520, with a height and width on the order of multiple meters.

Thus, with reference to FIGS. 1-5, arrangement 100 comprises orderprocessing component 110 that is operable to receive order 104identifying item 120. In some examples, item 120 comprises a perishablefood item. In some examples, item 120 is identifiable (to selectioncomponent 200) by barcode 214. In some examples, order processingcomponent 110 is further operable to identify a ripeness preference(e.g., from customer preferences 256 in order 104 and/or customerprofile 278). Arrangement 100 further comprises selection component 200comprising first hyperspectral sensor 204 that is operable to sensereflection 212 from item 120 and produce sensor output 206 based atleast on reflection 212. Arrangement 100 further comprises pickingmechanism 220, which is operable to divert item 120 to a selectionoutput 230 based at least on a determination to select item 120 forfulfillment of order 104.

In some examples, selection component 200 further comprises light source210 positioned to produce reflection 212 from item 120. In someexamples, light source 210 and first hyperspectral sensor 204 arecalibrated to produce a predetermined sensor output for a calibrationstandard. In some examples, selection component 200 further comprises atleast one additional sensor operable to contribute additional sensordata to the sensor output. The additional sensor is selected from thelist consisting of: second hyperspectral sensor 204 a positioned at adifferent viewing angle than first hyperspectral sensor 204 relative toitem 120, and optical camera 208. The additional sensor(s) contributeadditional sensor data to sensor output 206. In some examples, selectioncomponent 200 further comprises barcode reader 216, which is operable toidentify item 120.

Picking mechanism 220 is further operable to divert item 120 to disposaloutput 232 based at least on a determination to not select item 120 forfulfillment of order 104, for example if control component 250determines that item 120 may have passed its expiration. Pickingmechanism 220 is further operable to divert item 120 to inventory returnoutput 234, for example if control component 250 determines that item120 has not yet passed its expiration (and thus is still viable), but itexceeds the quantity identified in quantity value 108 of order 104. Insome examples, selection component 200 is stationary, and in someexamples, selection component 200 is mobile. In some examples, selectioncomponent 200 comprises a plurality of selection stations (e.g.,selection stations 202, 202 a, 202 b, and 202 c) and each selectionstation comprises a hyperspectral sensor (e.g., hyperspectral sensor204). Arrangement 100 further comprises disposal zone 130.

Arrangement 100 further comprises control component 250, which isoperable to receive order 104 from order processing component 110 andreceive sensor output 206 from selection component 200. Controlcomponent 250 is also operable to, based at least on identification ofitem 120, select a hyperspectral profile (e.g., selected hyperspectralprofile 254) from set of hyperspectral profiles 274. Control component250 is also operable to compare sensor output 206 with selectedhyperspectral profile 254 and, based at least on the comparison,determine whether to select item 120 for fulfillment of order 104. Insome examples, control component 250 is further operable to select thehyperspectral profile based at least on a ripeness preference, which maybe specified in order 104 and/or customer profile 278. In some examples,control component 250 is further operable to select the hyperspectralprofile based at least on an expected storage duration in order storagezone 500, for example as indicated in time estimate 114. In someexamples, control component 250 is co-located with selection component200.

In some examples, hyperspectral profile update component 262 is operableto update set of hyperspectral profiles 274 based at least on receivedfeedback 264 or environmental conditions (e.g., in historical data 280and/or localization data 284). In some examples, received feedback 264comprises customer feedback 116. In some examples, received feedback 264comprises feedback from an inspection process at disposal zone 130. Insome examples, hyperspectral profile update component 262 comprises MLcomponent 266.

Arrangement 100 further comprises transport component 400, which isoperable to transport item 120 to order storage zone 500. In someexamples, transport component is further operable to transport item 120to/from inventory zone 118 to selection component 200, to transport item120 to/from disposal zone 130, and/or to/from order storage zone 500.Control component 250 is further operable to instruct transportcomponent 400. In some examples, order storage zone 500 comprises anASRS 500 a.

FIG. 6 shows a flow chart 600 of exemplary operations associated witharrangement 100 (of FIG. 1), for example, automated food selection usinghyperspectral sensing. In some examples, some or all of the computeroperations described for flow chart 600 are performed ascomputer-executable instructions on computing node 800 (see FIG. 8).Flow chart 600 commences with operation 602, which includes receivingorder 104 identifying item 120. In some examples, item 120 comprises aperishable food item. Customer profile 278 is read during operation 604.Operation 606 includes identifying a ripeness preference, for examplefrom order 104 and/or customer profile 278. Item 120 is retrieved frominventory zone 118 into selection component 200 during operation 608,for example by transporting item 120 from inventory zone 118 toselection component 200.

In some examples, item 120 is identified by barcode 214 in operation610. Light source 210, which may be a calibrated light source, generatesillumination in operation 612, thereby producing reflection 212 fromitem 120 with light source 210. Operation 614 includes sensing, by firsthyperspectral sensor 204 of selection component 200, reflection 212 fromitem 120. Operation 616 includes producing, based at least on reflection212, sensor output 206. In some examples, operation 616 further includescontributing additional sensor data to sensor output 206 by at least oneadditional sensor. The additional sensor is at least one of: secondhyperspectral sensor 204 a positioned at a different viewing angle thanfirst hyperspectral sensor 204 relative to item 120, and optical camera208.

Operation 618 includes, based at least on identification of item 120,selecting a hyperspectral profile (e.g., selected hyperspectral profile254) from set of hyperspectral profiles 274. In some examples, selectingthe hyperspectral profile comprises selecting the hyperspectral profilebased at least on the ripeness preference. In some examples thehyperspectral profile is selected based at least on an expected storageduration in order storage zone 500. Operation 620 includes comparingsensor output 206 with selected hyperspectral profile 254. In someexamples, the result of this comparison returns an expected expirationdate (when item 120 has perished, or is below a quality threshold).Operation 622 includes, based at least on the comparison, determiningwhether to select item 120 for fulfillment of order 104. Operation 624includes control component 250 generating control instructions to sendto selection component 200 (e.g., whether to select item 120, returnitem 120 to inventory zone 118, or send item 120 to disposal zone 130).

Selection component 200 receives the determination, regarding item 120,in operation 626, and implements the three-way option indicated at 628:select for order 104, return to inventory zone 118, or send to disposalzone 130. If the determination is selection of item 120, operation 630includes, based at least on a determination to select item 120 forfulfillment of order 104, diverting item 120 to selection output 230.Transport component 400 is instructed (for example by control component250 or by selection component 200) to transport item 120 to orderstorage zone 500. Operation 634 includes the action of transporting item120 to order storage zone 500. In some examples, order storage zone 500comprises ASRS 500 a.

While item 120 is within order storage zone 500, decision operation 363monitors for a time-out condition, for example item 120 perishing. Sucha prediction could be made using the expected expiration date wasdetermined when sensor output 206 was compared with selectedhyperspectral profile 254, in operation 620. Upon expiration, flow chart600 moves to operation 662, described below. Otherwise, customer 102picks up (retrieves) item 120 in operation 638. Optionally, customer 102provides customer feedback 116 regarding whether item 120 was in therequested condition (e.g., ripe or pre-ripe), or whether item 120 hadalready expired. Customer feedback 116 is received in operation 640.

If the determination at 628 is to return item 120 to inventory zone 118,operation 650 includes, based at least on a determination to not selectitem 120 for fulfillment of order 104, diverting item 120 to inventoryreturn output 234. Transport component 400 is instructed in operation652, and operation 654 includes transporting item 120 from selectioncomponent 200 to inventory zone 118.

If the determination at 628 is to send item 120 to disposal zone forfurther inspection, operation 660 includes, based at least on adetermination to not select item 120 for fulfillment of order 104,diverting item 120 to disposal output 232. Transport component 400 isinstructed in operation 662, and operation 664 includes transportingitem 120 from selection component 200 to disposal zone 130. Aninspection process occurs at disposal zone 130 during operation 666, forexample human inspector 132 determining whether item 120 (now designateditem 120 d) is still viable or needs to be immediately disposed of(e.g., donated, offered for reduced price, or thrown into the trash).This determination is indicated to control component 250 via terminal134 as disposal feedback. Operation 668 includes control component 250receiving disposal feedback from disposal zone 130. Received feedback264 includes whichever of disposal feedback and customer feedback 116that is received by control component 250.

Environmental factors, such as heat and humidity (which may affectripening and perishing timelines), are monitored in operation 670.Supplier histories are monitored in operation 672, for example whichsupplier provides produce that spoils more quickly or ripens later afterdelivery. Such information may be stored in historical data, industrydata, and/or localization data 284. This data may be used for trainingML component 266, in operation 674. Operation 674 may also includetraining ML component 266 to better predict ripening and expiration,based on comparison of sensor output 206 with selected hyperspectralprofile 254 (e.g., during operation 620). Operation 676 includesupdating set of hyperspectral profiles 274 based at least on receivedfeedback 264 or environmental conditions. That is, the data used fortraining ML component 266 may also be used to update set ofhyperspectral profiles 274. In some examples, ML component 266 updatesset of hyperspectral profiles 274, based on its new training.

FIG. 7 shows a flow chart 700 of exemplary operations associated witharrangement 100 (of FIG. 1). In some examples, some or all of thecomputer operations described for flow chart 700 are performed ascomputer-executable instructions on computing node 800 (see FIG. 8).Flow chart 700 commences with operation 702, which includes receiving anorder identifying an item. Operation 704 includes sensing, by a firsthyperspectral sensor of a selection component, a reflection from theitem, producing, based at least on the reflection, a sensor output.Operation 706 includes based at least on identification of the item,selecting a hyperspectral profile from a set of hyperspectral profiles.Operation 708 includes comparing the sensor output with the selectedhyperspectral profile. Operation 710 includes based at least on thecomparison, determining whether to select the item for fulfillment ofthe order. Operation 712 includes based at least on a determination toselect the item for fulfillment of the order, diverting the item to aselection output. Operation 714 includes transporting the item to anorder storage zone.

Exemplary Operating Environment

FIG. 8 is a block diagram of an example computing node 800 forimplementing aspects disclosed herein and is designated generally ascomputing node 800. Computing node 800 is one example of a suitablecomputing environment and is not intended to suggest any limitation asto the scope of use or functionality of the invention. Neither shouldthe computing node 800 be interpreted as having any dependency orrequirement relating to any one or combination of components/modulesillustrated. The examples and embodiments disclosed herein may bedescribed in the general context of computer code or machine-useableinstructions, including computer-executable instructions such as programcomponents, being executed by a computer or other machine, such as apersonal data assistant or other handheld device. Generally, programcomponents including routines, programs, objects, components, datastructures, and the like, refer to code that performs particular tasks,or implement particular abstract data types. The disclosed examples maybe practiced in a variety of system configurations, including personalcomputers, laptops, smart phones, mobile tablets, hand-held devices,consumer electronics, specialty computing nodes, etc. The disclosedexamples may also be practiced in distributed computing environments,where tasks are performed by remote-processing devices that are linkedthrough network 830 (e.g., a communications network).

Computing node 800 includes a bus 810 that directly or indirectlycouples the following devices: memory 812, one or more processors 814,one or more presentation components 816, input/output (I/O) ports 818,I/O components 820, a power supply 822, and a network component 824.Computing node 800 should not be interpreted as having any dependency orrequirement related to any single component or combination of componentsillustrated therein. While computing node 800 is depicted as a seeminglysingle device, multiple computing nodes 800 may work together and sharethe depicted device resources. That is, one or more computer storagedevices having computer-executable instructions stored thereon mayperform operations disclosed herein. For example, memory 812 may bedistributed across multiple devices, processor(s) 814 may provide housedon different devices, and so on.

Bus 810 represents what may be one or more busses (such as an addressbus, data bus, or a combination thereof). Although the various blocks ofFIG. 8 are shown with lines for the sake of clarity, delineating variouscomponents can be accomplished with various other schemes. For example,a presentation component such as a display device can also be classifiedas an I/O component. Additionally, processors have internal memory.Thus, the diagram of FIG. 8 is merely illustrative of an exemplarycomputing node that can be used in connection with one or moreembodiments. Distinction is not made between such categories as“workstation,” “server,” “laptop,” “hand-held device,” etc., as all arecontemplated within the scope of FIG. 8 and the references herein to a“computing node” or a “computing device.” Memory 812 may include any ofthe computer-readable media discussed herein. Memory 812 is used tostore and access data 812 a and instructions 812 b operable to carry outthe various operations disclosed herein. In some examples, memory 812includes computer storage media in the form of volatile and/ornonvolatile memory, removable or non-removable memory, data disks invirtual environments, or a combination thereof.

Processor(s) 814 may include any quantity of processing units that readdata from various entities, such as memory 812 or I/O components 820.Specifically, processor(s) 814 are programmed to executecomputer-executable instructions for implementing aspects of thedisclosure. The instructions may be performed by the processor, bymultiple processors within the computing node 800, or by a processorexternal to the client computing node 800. In some examples, theprocessor(s) 814 are programmed to execute instructions such as thoseillustrated in the flowcharts discussed below and depicted in theaccompanying drawings. Moreover, in some examples, the processor(s) 814represent an implementation of analog techniques to perform theoperations described herein. For example, the operations may beperformed by an analog client computing node 800 and/or a digital clientcomputing node 800.

Presentation component(s) 816 present data indications to a user orother device. Exemplary presentation components include a displaydevice, speaker, printing component, vibrating component, etc. Oneskilled in the art will understand and appreciate that computer data maybe presented in a number of ways, such as visually in a graphical userinterface (GUI), audibly through speakers, wirelessly among multiplecomputing nodes 800, across a wired connection, or in other ways. Ports818 allow computing node 800 to be logically coupled to other devicesincluding I/O components 820, some of which may be built in. Example I/Ocomponents 820 include, for example but without limitation, amicrophone, keyboard, mouse, joystick, game pad, satellite dish,scanner, printer, wireless device, etc.

In some examples, the network component 824 includes a network interfacecard and/or computer-executable instructions (e.g., a driver) foroperating the network interface card. Communication between thecomputing node 800 and other devices may occur using any protocol ormechanism over any wired or wireless connection. In some examples, thenetwork component 824 is operable to communicate data over public,private, or hybrid (public and private) network 830 using a transferprotocol, between devices wirelessly using short range communicationtechnologies (e.g., near-field communication (NFC), Bluetooth® brandedcommunications, or the like), or a combination thereof. Networkcomponent 824 communicates over wireless communication link 826 and/or awired communication link 826 a to a cloud resource 828 across network830. Various different examples of communication links 826 and 826 ainclude a wireless connection, a wired connection, and/or a dedicatedlink, and in some examples, at least a portion is routed through theinternet.

Although described in connection with an example computing node 800,examples of the disclosure are capable of implementation with numerousother general-purpose or special-purpose computing system environments,configurations, or devices. Examples of well-known computing systems,environments, and/or configurations that may be suitable for use withaspects of the disclosure include, but are not limited to, smart phones,mobile tablets, mobile computing nodes, personal computers, servercomputers, hand-held or laptop devices, multiprocessor systems, gamingconsoles, microprocessor-based systems, set top boxes, programmableconsumer electronics, mobile telephones, mobile computing and/orcommunication devices in wearable or accessory form factors (e.g.,watches, glasses, headsets, or earphones), network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, virtual reality (VR) devices,holographic device, and the like. Such systems or devices may acceptinput from the user in any way, including from input devices such as akeyboard or pointing device, via gesture input, proximity input (such asby hovering), and/or via voice input.

Examples of the disclosure may be described in the general context ofcomputer-executable instructions, such as program modules, executed byone or more computers or other devices in software, firmware, hardware,or a combination thereof. The computer-executable instructions may beorganized into one or more computer-executable components or modules.Generally, program modules include, but are not limited to, routines,programs, objects, components, and data structures that performparticular tasks or implement particular abstract data types. Aspects ofthe disclosure may be implemented with any number and organization ofsuch components or modules. For example, aspects of the disclosure arenot limited to the specific computer-executable instructions or thespecific components or modules illustrated in the figures and describedherein. Other examples of the disclosure may include differentcomputer-executable instructions or components having more or lessfunctionality than illustrated and described herein. In examplesinvolving a general-purpose purpose computer, aspects of the disclosuretransform the general-purpose computer into a special-purpose computingdevice or computing node when configured to execute the instructionsdescribed herein.

By way of example and not limitation, computer readable media comprisecomputer storage media and communication media. Computer storage mediainclude volatile and nonvolatile, removable and non-removable memoryimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules, orthe like. Computer storage media are tangible and mutually exclusive tocommunication media. Computer storage media are implemented in hardwareand exclude carrier waves and propagated signals. Computer storage mediafor purposes of this disclosure are not signals per se. Exemplarycomputer storage media include hard disks, flash drives, solid-statememory, phase change random-access memory (PRAM), static random-accessmemory (SRAM), dynamic random-access memory (DRAM), other types ofrandom-access memory (RAM), read-only memory (ROM), electricallyerasable programmable read-only memory (EEPROM), flash memory or othermemory technology, compact disk read-only memory (CD-ROM), digitalversatile disks (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other non-transmission medium that can be used to storeinformation for access by a computing device. In contrast, communicationmedia typically embody computer readable instructions, data structures,program modules, or the like in a modulated data signal such as acarrier wave or other transport mechanism and include any informationdelivery media.

Exemplary Operating Methods and Systems

An exemplary system for automated food selection using hyperspectralsensing comprises: an order processing component operable to receive anorder identifying an item; a selection component comprising: a firsthyperspectral sensor operable to sense a reflection from the item andproduce a sensor output based at least on the reflection; and a pickingmechanism; a control component operable to: receive the order from theorder processing component; receive the sensor output from the selectioncomponent; based at least on identification of the item, select ahyperspectral profile from a set of hyperspectral profiles; compare thesensor output with the selected hyperspectral profile; and based atleast on the comparison, determine whether to select the item forfulfillment of the order, wherein the picking mechanism is operable todivert the item to a selection output based at least on a determinationto select the item for fulfillment of the order; and a transportcomponent operable to transport the item to an order storage zone.

An exemplary method of automated food selection using hyperspectralsensing comprises: receiving an order identifying an item; sensing, by afirst hyperspectral sensor of a selection component, a reflection fromthe item, producing, based at least on the reflection, a sensor output;based at least on identification of the item, selecting a hyperspectralprofile from a set of hyperspectral profiles; comparing the sensoroutput with the selected hyperspectral profile; based at least on thecomparison, determining whether to select the item for fulfillment ofthe order; based at least on a determination to select the item forfulfillment of the order, diverting the item to a selection output; andtransporting the item to an order storage zone.

An exemplary computer storage device has computer-executableinstructions stored thereon for automated food selection usinghyperspectral sensing, which, on execution by a computer, cause thecomputer to perform operations comprising: receiving an orderidentifying an item, wherein the item comprises a perishable food item;receiving sensor output from a hyperspectral sensor, wherein the sensoroutput is based at least on a reflection from the item; based at leaston identification of the item, selecting a hyperspectral profile from aset of hyperspectral profiles; comparing the sensor output with theselected hyperspectral profile; based at least on the comparison,determining whether to select the item for fulfillment of the order;based at least on a determination to select the item for fulfillment ofthe order, diverting the item to a selection output and controllingtransport of the item to an ASRS; based at least on a determination tonot select the item for fulfillment of the order, diverting the item toa disposal output and controlling transport of the item to a disposalzone; and updating the set of hyperspectral profiles based at least onreceived feedback or environmental conditions.

Alternatively, or in addition to the other examples described herein,examples include any combination of the following:

-   -   the selection component further comprises a light source        positioned to produce the reflection from the item;    -   producing the reflection from the item with a light source;    -   the selection component further comprises at least one        additional sensor operable to contribute additional sensor data        to the sensor output;    -   contributing additional sensor data to the sensor output by at        least one additional sensor;    -   the additional sensor is selected from the list consisting of: a        second hyperspectral sensor positioned at a different viewing        angle than the first hyperspectral sensor relative to the item,        and an optical camera;    -   the transport component is further operable to transport the        item from an inventory zone to the selection component;    -   transporting the item from an inventory zone to the selection        component;    -   the transport component is further operable to transport the        item to the disposal zone;    -   the transport component is further operable to transport the        selection component with the item to the order storage zone;    -   the order storage zone comprises an ASRS;    -   the order processing component is further operable to identify a        ripeness preference;    -   identifying a ripeness preference;    -   the control component is further operable to select the        hyperspectral profile based at least on the ripeness preference;    -   selecting the hyperspectral profile comprises selecting the        hyperspectral profile based at least on the ripeness preference;    -   a disposal zone;    -   the picking mechanism is further operable to divert the item to        a disposal output based at least on a determination to not        select the item for fulfillment of the order;    -   based at least on a determination to not select the item for        fulfillment of the order, diverting the item to a disposal        output;    -   a hyperspectral profile update component operable to update the        set of hyperspectral profiles based at least on received        feedback or environmental conditions;    -   updating the set of hyperspectral profiles based at least on        received feedback or environmental conditions;    -   the item comprises a perishable food item;    -   the selection component further comprises a barcode reader        operable to identify the item;    -   the item is identified by a barcode;    -   the picking mechanism is further operable to divert the item to        an inventory return output;    -   the light source and the first hyperspectral sensor are        calibrated to produce a predetermined sensor output for a        calibration standard;    -   the selection component is stationary;    -   the selection component comprises a plurality of selection        stations and each selection station comprises a hyperspectral        sensor;    -   the control component is further operable to instruct the        transport component;    -   the ripeness preference is specified in the order;    -   the ripeness preference is specified in a customer profile;    -   the received feedback comprises customer feedback;    -   the received feedback comprises feedback from an inspection        process at the disposal zone;    -   the control component is further operable to select the        hyperspectral profile based at least on an expected storage        duration in the order storage zone;    -   the control component is co-located with the selection        component; and    -   the hyperspectral profile update component comprises an ML        component.

The order of execution or performance of the operations in examples ofthe disclosure illustrated and described herein may not be essential,and thus may be performed in different sequential manners in variousexamples. For example, it is contemplated that executing or performing aparticular operation before, contemporaneously with, or after anotheroperation is within the scope of aspects of the disclosure. Whenintroducing elements of aspects of the disclosure or the examplesthereof, the articles “a,” “an,” “the,” and “said” are intended to meanthat there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements. Theterm “exemplary” is intended to mean “an example of” The phrase “one ormore of the following: A, B, and C” means “at least one of A and/or atleast one of B and/or at least one of C.”

Having described aspects of the disclosure in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of aspects of the disclosure as defined in theappended claims. As various changes could be made in the aboveconstructions, products, and methods without departing from the scope ofaspects of the disclosure, it is intended that all matter contained inthe above description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense. While thedisclosure is susceptible to various modifications and alternativeconstructions, certain illustrated examples thereof are shown in thedrawings and have been described above in detail. It should beunderstood, however, that there is no intention to limit the disclosureto the specific forms disclosed, but on the contrary, the intention isto cover all modifications, alternative constructions, and equivalentsfalling within the spirit and scope of the disclosure.

What is claimed is:
 1. A system for automated food selection usinghyperspectral sensing, the system comprising: an order processingcomponent operable to receive an order identifying an item; a selectioncomponent comprising: a first hyperspectral sensor operable to sense areflection from the item and produce a sensor output based at least onthe reflection; and a picking mechanism; a control component operableto: receive the order from the order processing component; receive thesensor output from the selection component; based at least onidentification of the item, select a hyperspectral profile from a set ofhyperspectral profiles; compare the sensor output with the selectedhyperspectral profile; and based at least on the comparison, determinewhether to select the item for fulfillment of the order, wherein thepicking mechanism is operable to divert the item to a selection outputbased at least on a determination to select the item for fulfillment ofthe order.
 2. The system of claim 1, wherein the selection componentfurther comprises a light source positioned to produce the reflectionfrom the item.
 3. The system of claim 1, wherein the selection componentfurther comprises at least one additional sensor operable to contributeadditional sensor data to the sensor output, and wherein the additionalsensor is selected from the list consisting of: a second hyperspectralsensor positioned at a different viewing angle than the firsthyperspectral sensor relative to the item, and an optical camera.
 4. Thesystem of claim 1, further comprising: a transport component operable totransport the item to an order storage zone, wherein the transportcomponent is further operable to transport the item from an inventoryzone to the selection component.
 5. The system of claim 4, wherein thetransport component is further operable to transport the selectioncomponent with the item to the order storage zone.
 6. The system ofclaim 4, wherein the order storage zone comprises an automated storageand retrieval system (ASRS).
 7. The system of claim 1, wherein the orderprocessing component is further operable to identify a ripenesspreference and wherein the control component is further operable toselect the hyperspectral profile based at least on the ripenesspreference.
 8. The system of claim 1, further comprising: a disposalzone, wherein the picking mechanism is further operable to divert theitem to a disposal output based at least on a determination to notselect the item for fulfillment of the order, and wherein the transportcomponent is further operable to transport the item to the disposalzone.
 9. The system of claim 1, further comprising: a hyperspectralprofile update component operable to update the set of hyperspectralprofiles based at least on received feedback or environmentalconditions.
 10. The system of claim 1, wherein the item comprises aperishable food item.
 11. A method of automated food selection usinghyperspectral sensing, the method comprising: receiving an orderidentifying an item; sensing, by a first hyperspectral sensor of aselection component, a reflection from the item, producing, based atleast on the reflection, a sensor output; based at least onidentification of the item, selecting a hyperspectral profile from a setof hyperspectral profiles; comparing the sensor output with the selectedhyperspectral profile; based at least on the comparison, determiningwhether to select the item for fulfillment of the order; and based atleast on a determination to select the item for fulfillment of theorder, diverting the item to a selection output.
 12. The method of claim11, further comprising: producing the reflection from the item with alight source.
 13. The method of claim 11, further comprising:contributing additional sensor data to the sensor output by at least oneadditional sensor, wherein the additional sensor is selected from thelist consisting of: a second hyperspectral sensor positioned at adifferent viewing angle than the first hyperspectral sensor relative tothe item, and an optical camera.
 14. The method of claim 11, furthercomprising: transporting the item from an inventory zone to theselection component.
 15. The method of claim 11, wherein the orderstorage zone comprises an automated storage and retrieval system (ASRS).16. The method of claim 11, further comprising: identifying a ripenesspreference, and wherein selecting the hyperspectral profile comprisesselecting the hyperspectral profile based at least on the ripenesspreference.
 17. The method of claim 11, further comprising: based atleast on a determination to not select the item for fulfillment of theorder, diverting the item to a disposal output.
 18. The method of claim11, further comprising: updating the set of hyperspectral profiles basedat least on received feedback or environmental conditions.
 19. Themethod of claim 11, wherein the item comprises a perishable food item.20. One or more computer storage devices having computer-executableinstructions stored thereon, which, on execution by a computer, causethe computer to perform operations comprising: receiving an orderidentifying an item, wherein the item comprises a perishable food item;receiving sensor output from a hyperspectral sensor, wherein the sensoroutput is based at least on a reflection from the item; based at leaston identification of the item, selecting a hyperspectral profile from aset of hyperspectral profiles; comparing the sensor output with theselected hyperspectral profile; based at least on the comparison,determining whether to select the item for fulfillment of the order;based at least on a determination to select the item for fulfillment ofthe order, diverting the item to a selection output and controllingtransport of the item to an automated storage and retrieval system(ASRS); based at least on a determination to not select the item forfulfillment of the order, diverting the item to a disposal output andcontrolling transport of the item to a disposal zone; and updating theset of hyperspectral profiles based at least on received feedback orenvironmental conditions.